Deploy opersight suite

Deploy opersight-ai-fastapi (port 8006) and opersight-copilot-server (port 8007) on msi-laptop-3 using the generic hydrotrek-ai-fastapi and hydrotrek-copilot-server repos. Both services share Docker images that are built once and re-used by any future suite (swmm, flooding, etc.).

This is a step-by-step guide for deploying Opersight first as the example app. The same pattern can be reused later for swmm and flooding with different names, ports, env files, and Nginx paths.

Path notes:

Workstation paths such as c:/Git/gqc/... are examples from the current Windows machine. Adjust them if your checkout is somewhere else.
Server paths such as /opt/apis/... and /etc/opersight-stack/... follow the msi-laptop-3 docs. Verify the actual server layout before copying commands.
Commands marked "server" should be run over SSH on msi-laptop-3, not from Windows PowerShell.

Architecture

Browser
  │
  ▼
Nginx :443  (www.hydrotrek.org → 74.83.141.228)
  ├── /gqc/api/...            → existing DWD stack (8001–8003)
  ├── /opersight/api/fastapi/ → 127.0.0.1:8006 → opersight-ai-fastapi :8080
  └── /opersight/api/copilotkit → 127.0.0.1:8007 → opersight-copilot-server :8082

Docker network (opersight-stack — server-to-server):
  opersight-copilot-server ──AGUI_AGENT_URL──► opersight-ai-fastapi:8080/agent

Public URLs (after Nginx and SSL):

Endpoint	URL
FastAPI health	`https://www.hydrotrek.org/opersight/api/fastapi/health`
FastAPI Swagger	`https://www.hydrotrek.org/opersight/api/fastapi/docs`
CopilotKit runtime	`https://www.hydrotrek.org/opersight/api/copilotkit`

Port allocation

Service	Container port	Host bind (`127.0.0.1`)	Nginx path
`opersight-ai-fastapi`	8080	8006	`/opersight/api/fastapi/`
`opersight-copilot-server`	8082	8007	`/opersight/api/copilotkit`

Prerequisites

DWD stack already running (deploy keys pattern established)
Docker + compose installed on server (08-docker-install-and-users.md)
apisvc user exists; your login user is in the apisvc group
OPENAI_API_KEY ready for opersight

Step 1 — Add Dockerfile to `hydrotrek-ai-fastapi`

The hydrotrek-ai-fastapi repo has no Dockerfile. Add the following file, commit, and push before running server commands.

File: c:/Git/gqc/hydrotrek-ai-fastapi/Dockerfile

If the repo is checked out somewhere else, create the file at the root of that repo.

# hydrotrek-ai-fastapi — generic HydroTrek AI agent backend
FROM python:3.11-slim-bookworm

WORKDIR /app

ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1 \
    PIP_NO_CACHE_DIR=1 \
    PORT=8080

RUN apt-get update \
    && apt-get install -y --no-install-recommends curl \
    && rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN pip install --upgrade pip \
    && pip install -r requirements.txt

COPY . .

RUN useradd --create-home --uid 10001 appuser \
    && chown -R appuser:appuser /app
USER appuser

EXPOSE 8080

HEALTHCHECK --interval=30s --timeout=5s --start-period=60s --retries=3 \
    CMD curl -f http://127.0.0.1:8080/health || exit 1

ENV GUNICORN_WORKERS=2
CMD ["gunicorn", "--config", "gunicorn.conf.py", "main:app"]

Commit and push from the repo root:

# Windows path shown as an example. Use your actual local repo path.
cd c:/Git/gqc/hydrotrek-ai-fastapi
git add Dockerfile
git commit -m "Add Dockerfile for server deployment"
git push

Step 2 — Create Opersight template files

Create the following files in this repo under it/MSI Laptop 3 Server/. Commit and push so the server operator can reference them.

`templates/docker/hydrotrek-opersight-suite/docker-compose.yml`

Local dev compose (submodule build contexts — assumes clones alongside this directory).

# Local dev — run from hydrotrek-opersight-suite/
# Copy env/*.env.example → env/*.env and fill secrets

services:
  opersight-ai-fastapi:
    build:
      context: ../hydrotrek-ai-fastapi
      dockerfile: Dockerfile
    image: hydrotrek-ai-fastapi:latest
    ports:
      - "127.0.0.1:8006:8080"
    env_file:
      - env/opersight-ai-fastapi.env
    environment:
      PORT: "8080"
      GUNICORN_WORKERS: "2"
      GUNICORN_TIMEOUT: "120"
    networks:
      - opersight-stack
    restart: unless-stopped

  opersight-copilot-server:
    build:
      context: ../hydrotrek-copilot-server
      dockerfile: Dockerfile
    image: hydrotrek-copilot-server:latest
    ports:
      - "127.0.0.1:8007:8082"
    env_file:
      - env/opersight-copilot-server.env
    environment:
      PORT: "8082"
      AGUI_AGENT_URL: http://opersight-ai-fastapi:8080/agent
    networks:
      - opersight-stack
    restart: unless-stopped

networks:
  opersight-stack:
    name: opersight-stack
    driver: bridge

`templates/docker/hydrotrek-opersight-suite/docker-compose.prod.yml`

Production reference — copy this to /etc/opersight-stack/docker-compose.yml on the server.

# Production — copy to /etc/opersight-stack/docker-compose.yml on the server
#   sudo cp docker-compose.prod.yml /etc/opersight-stack/docker-compose.yml
#
# Requires flat clones under /opt/apis/ and env files under /etc/<service>/.

services:
  opersight-ai-fastapi:
    build:
      context: /opt/apis/hydrotrek-ai-fastapi
      dockerfile: Dockerfile
    image: hydrotrek-ai-fastapi:latest
    ports:
      - "127.0.0.1:8006:8080"
    env_file:
      - /etc/opersight-ai-fastapi/opersight-ai-fastapi.env
    environment:
      PORT: "8080"
      GUNICORN_WORKERS: "2"
      GUNICORN_TIMEOUT: "120"
    networks:
      - opersight-stack
    restart: unless-stopped

  opersight-copilot-server:
    build:
      context: /opt/apis/hydrotrek-copilot-server
      dockerfile: Dockerfile
    image: hydrotrek-copilot-server:latest
    ports:
      - "127.0.0.1:8007:8082"
    env_file:
      - /etc/opersight-copilot-server/opersight-copilot-server.env
    environment:
      PORT: "8082"
      AGUI_AGENT_URL: http://opersight-ai-fastapi:8080/agent
    networks:
      - opersight-stack
    restart: unless-stopped

networks:
  opersight-stack:
    name: opersight-stack
    driver: bridge

`templates/docker/hydrotrek-opersight-suite/env/opersight-ai-fastapi.env.example`

# Copy to /etc/opersight-ai-fastapi/opersight-ai-fastapi.env on server

OPENAI_API_KEY=
OPENAI_CHAT_MODEL_ID=gpt-4o-mini
OPENAI_RESPONSES_STORE=false

# Agent system prompt for opersight
# AGENT_INSTRUCTIONS=You are a helpful Opersight assistant...

# Public URL prefix for Swagger UI behind Nginx
ROOT_PATH=/opersight/api/fastapi

# Extra CORS origins beyond localhost defaults
# CORS_EXTRA=https://opersight.hydrotrek.org

`templates/docker/hydrotrek-opersight-suite/env/opersight-copilot-server.env.example`

# Copy to /etc/opersight-copilot-server/opersight-copilot-server.env on server

# Must match agent="..." in the opersight frontend CopilotKit component
AGENT_ID=opersight_agent

# AGUI_AGENT_URL is set in docker-compose — do not override unless debugging
# AGUI_AGENT_URL=http://opersight-ai-fastapi:8080/agent

# Extra CORS origins beyond localhost defaults
# CORS_EXTRA=https://opersight-frontend.azurestaticapps.net

`templates/nginx/opersight-gqc-addition.conf`

Upstream blocks and location blocks to add to the existing /etc/nginx/sites-available/dwd-gqc file.

Why there are two FastAPI-looking paths:

location /opersight/api/fastapi/ handles real FastAPI routes like /health and /docs.
location = /opersight/api/fastapi only redirects the no-trailing-slash URL to the slash form.

CopilotKit does not need two separate blocks here. One prefix block, location /opersight/api/copilotkit, covers both the exact runtime URL and any subpaths under it.

# Opersight suite upstreams.
# Add these before the first server{} block in /etc/nginx/sites-available/dwd-gqc.

upstream opersight_ai_fastapi {
    server 127.0.0.1:8006;
    keepalive 32;
}

upstream opersight_copilot_server {
    server 127.0.0.1:8007;
    keepalive 16;
}

# Opersight location blocks.
# Add these inside the existing HTTPS server{} block:
#   server { listen 443 ssl ... }

location /opersight/api/fastapi/ {
    proxy_pass http://opersight_ai_fastapi/;
    proxy_http_version 1.1;

    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_read_timeout 300s;
    proxy_send_timeout 300s;

    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
}

location = /opersight/api/fastapi {
    return 301 /opersight/api/fastapi/;
}

location /opersight/api/copilotkit {
    proxy_pass http://opersight_copilot_server/api/copilotkit;
    proxy_http_version 1.1;

    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_read_timeout 300s;
    proxy_send_timeout 300s;

    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
}

Step 3 — Server: Create directories

SSH into msi-laptop-3. All commands run as your normal login user (e.g. gqc).

# Shared repo dirs (skip if already created for a previous suite)
sudo mkdir -p /opt/apis/hydrotrek-ai-fastapi
sudo mkdir -p /opt/apis/hydrotrek-copilot-server
sudo chown apisvc:apisvc /opt/apis/hydrotrek-ai-fastapi /opt/apis/hydrotrek-copilot-server

# Opersight-specific env dirs
sudo mkdir -p /etc/opersight-ai-fastapi
sudo chown root:apisvc /etc/opersight-ai-fastapi
sudo chmod 750 /etc/opersight-ai-fastapi

sudo mkdir -p /etc/opersight-copilot-server
sudo chown root:apisvc /etc/opersight-copilot-server
sudo chmod 750 /etc/opersight-copilot-server

# Stack compose dir
sudo mkdir -p /etc/opersight-stack

# Log dirs (optional — Docker logs go to stdout by default)
sudo mkdir -p /var/log/opersight-ai-fastapi /var/log/opersight-copilot-server
sudo chown apisvc:apisvc /var/log/opersight-ai-fastapi /var/log/opersight-copilot-server

Step 4 — Server: Set up deploy keys

Generate one read-only deploy key per shared repo. Skip a repo's key if the clone already exists.

`hydrotrek-ai-fastapi` deploy key

API=hydrotrek-ai-fastapi
GITHUB_ORG=gqc

ssh-keygen -t ed25519 \
  -f ~/.ssh/${API}_deploy \
  -C "msi-laptop-3-${API}" \
  -N ""

chmod 600 ~/.ssh/${API}_deploy
chmod 644 ~/.ssh/${API}_deploy.pub

echo "=== Add this public key to GitHub ==="
cat ~/.ssh/${API}_deploy.pub

Add the public key at https://github.com/gqc/hydrotrek-ai-fastapi/settings/keys:

Title: MSI Laptop 3 — hydrotrek-ai-fastapi
Allow write access: unchecked

Append SSH config:

cat >> ~/.ssh/config << 'EOF'

Host github.com-hydrotrek-ai-fastapi
    HostName github.com
    User git
    IdentityFile ~/.ssh/hydrotrek-ai-fastapi_deploy
    IdentitiesOnly yes
EOF
chmod 600 ~/.ssh/config

Test:

ssh -T git@github.com-hydrotrek-ai-fastapi
# Expected: Hi gqc/hydrotrek-ai-fastapi! You've successfully authenticated...

`hydrotrek-copilot-server` deploy key

API=hydrotrek-copilot-server
GITHUB_ORG=gqc

ssh-keygen -t ed25519 \
  -f ~/.ssh/${API}_deploy \
  -C "msi-laptop-3-${API}" \
  -N ""

chmod 600 ~/.ssh/${API}_deploy
chmod 644 ~/.ssh/${API}_deploy.pub

echo "=== Add this public key to GitHub ==="
cat ~/.ssh/${API}_deploy.pub

Add at https://github.com/gqc/hydrotrek-copilot-server/settings/keys:

Title: MSI Laptop 3 — hydrotrek-copilot-server
Allow write access: unchecked

Append SSH config:

cat >> ~/.ssh/config << 'EOF'

Host github.com-hydrotrek-copilot-server
    HostName github.com
    User git
    IdentityFile ~/.ssh/hydrotrek-copilot-server_deploy
    IdentitiesOnly yes
EOF
chmod 600 ~/.ssh/config

Test:

ssh -T git@github.com-hydrotrek-copilot-server

Step 5 — Server: Clone repos

git clone git@github.com-hydrotrek-ai-fastapi:gqc/hydrotrek-ai-fastapi.git \
  /opt/apis/hydrotrek-ai-fastapi

git clone git@github.com-hydrotrek-copilot-server:gqc/hydrotrek-copilot-server.git \
  /opt/apis/hydrotrek-copilot-server

sudo chown -R apisvc:apisvc /opt/apis/hydrotrek-ai-fastapi
sudo chown -R apisvc:apisvc /opt/apis/hydrotrek-copilot-server
sudo chmod -R g+rwX /opt/apis/hydrotrek-ai-fastapi
sudo chmod -R g+rwX /opt/apis/hydrotrek-copilot-server

Verify:

ls /opt/apis/hydrotrek-ai-fastapi/Dockerfile    # must exist; see Step 1
ls /opt/apis/hydrotrek-copilot-server/Dockerfile

Step 6 — Server: Build Docker images

Build once. All future suites (swmm, flooding) reuse these images.

docker build -t hydrotrek-ai-fastapi:latest /opt/apis/hydrotrek-ai-fastapi
docker build -t hydrotrek-copilot-server:latest /opt/apis/hydrotrek-copilot-server

Confirm images exist:

docker images | grep hydrotrek

Step 7 — Server: Create env files

Copy env templates from the general repo (or from this doc) and fill in secrets.

# FastAPI env
sudo cp /opt/apis/hydrotrek-ai-fastapi/.env.example \
        /etc/opersight-ai-fastapi/opersight-ai-fastapi.env
sudo chmod 640 /etc/opersight-ai-fastapi/opersight-ai-fastapi.env
sudo chown root:apisvc /etc/opersight-ai-fastapi/opersight-ai-fastapi.env
sudo nano /etc/opersight-ai-fastapi/opersight-ai-fastapi.env

Minimum values to set in the FastAPI env file:

OPENAI_API_KEY=<your-key>
OPENAI_CHAT_MODEL_ID=gpt-4o-mini
OPENAI_RESPONSES_STORE=false
ROOT_PATH=/opersight/api/fastapi
# AGENT_INSTRUCTIONS=You are a helpful Opersight assistant...

# Copilot server env
sudo cp /opt/apis/hydrotrek-copilot-server/.env.example \
        /etc/opersight-copilot-server/opersight-copilot-server.env
sudo chmod 640 /etc/opersight-copilot-server/opersight-copilot-server.env
sudo chown root:apisvc /etc/opersight-copilot-server/opersight-copilot-server.env
sudo nano /etc/opersight-copilot-server/opersight-copilot-server.env

Minimum values to set in the copilot server env file:

AGENT_ID=opersight_agent
# AGUI_AGENT_URL is set in docker-compose — leave commented out

Step 8 — Server: Place compose file

Choose one of these options.

Option A: copy the template from the general repo if that repo is also present on the server. Replace /path/to/general with the actual server path.

GENERAL_REPO=/path/to/general   # Example only. Verify this path on msi-laptop-3.

sudo cp "$GENERAL_REPO/it/MSI Laptop 3 Server/templates/docker/hydrotrek-opersight-suite/docker-compose.prod.yml" \
        /etc/opersight-stack/docker-compose.yml

Option B: create the file directly from this guide. Copy the contents from the docker-compose.prod.yml section above into /etc/opersight-stack/docker-compose.yml.

sudo nano /etc/opersight-stack/docker-compose.yml

Verify the file is in place:

cat /etc/opersight-stack/docker-compose.yml

Step 9 — Server: Bring up containers

cd /etc/opersight-stack
sudo docker compose up -d

Smoke-test both services (direct, bypassing Nginx):

# FastAPI health — expect JSON with agent_enabled: true
curl -s http://127.0.0.1:8006/health | python3 -m json.tool

# Copilot server — expect stub status page (HTTP 200)
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8007/

Check logs if something is wrong:

sudo docker compose -f /etc/opersight-stack/docker-compose.yml logs -f --tail=50
sudo docker compose -f /etc/opersight-stack/docker-compose.yml logs opersight-ai-fastapi --tail=50

Confirm containers are bound to 127.0.0.1 only (not 0.0.0.0):

ss -tlnp | grep -E '8006|8007'
# Should show 127.0.0.1:8006 and 127.0.0.1:8007 only

Step 10 — Server: Add Nginx routes

Append the opersight upstream and location blocks from templates/nginx/opersight-gqc-addition.conf into the existing Nginx config.

# Open the existing dwd-gqc config
sudo nano /etc/nginx/sites-available/dwd-gqc

Inside that file:

Paste the two upstream opersight_* blocks before the first server { block
Paste the three location blocks inside the server { listen 443 ssl ... } block, before the closing }

Test and reload:

sudo nginx -t
sudo systemctl reload nginx

Verify through Nginx (HTTPS):

curl -s https://www.hydrotrek.org/opersight/api/fastapi/health
curl -s -X OPTIONS https://www.hydrotrek.org/opersight/api/copilotkit \
  -H "Origin: http://localhost:3000" -I

Step 11 — Server: Add systemd boot unit

Install a systemd unit so the opersight stack restarts automatically on reboot.

sudo tee /etc/systemd/system/opersight-stack.service > /dev/null << 'EOF'
[Unit]
Description=opersight-stack (Docker)
Documentation=file:///etc/opersight-stack/docker-compose.yml
After=docker.service network-online.target
Requires=docker.service
Wants=network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/etc/opersight-stack

ExecStart=/usr/bin/docker compose -f /etc/opersight-stack/docker-compose.yml up -d --remove-orphans
ExecStop=/usr/bin/docker compose -f /etc/opersight-stack/docker-compose.yml down
ExecReload=/usr/bin/docker compose -f /etc/opersight-stack/docker-compose.yml up -d --remove-orphans

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable opersight-stack
sudo systemctl status opersight-stack

Step 12 — Verify the deployment

Run from the server, then confirm from a browser.

# FastAPI through Nginx
curl -s https://www.hydrotrek.org/opersight/api/fastapi/health | python3 -m json.tool

# Swagger UI reachable
curl -s -o /dev/null -w "%{http_code}\n" \
  https://www.hydrotrek.org/opersight/api/fastapi/docs

# CopilotKit preflight (CORS)
curl -s -X OPTIONS https://www.hydrotrek.org/opersight/api/copilotkit \
  -H "Origin: http://localhost:3000" \
  -H "Access-Control-Request-Method: POST" -I

# Root still returns 404 (no leakage)
curl -s -o /dev/null -w "%{http_code}\n" https://www.hydrotrek.org/

Routine update procedure

When hydrotrek-ai-fastapi or hydrotrek-copilot-server is updated on GitHub:

# Pull latest code
cd /opt/apis/hydrotrek-ai-fastapi && git pull
cd /opt/apis/hydrotrek-copilot-server && git pull

# Rebuild images
docker build -t hydrotrek-ai-fastapi:latest /opt/apis/hydrotrek-ai-fastapi
docker build -t hydrotrek-copilot-server:latest /opt/apis/hydrotrek-copilot-server

# Recreate opersight containers with the new images (zero-downtime swap per service)
sudo docker compose -f /etc/opersight-stack/docker-compose.yml up -d \
  --no-deps opersight-ai-fastapi

sudo docker compose -f /etc/opersight-stack/docker-compose.yml up -d \
  --no-deps opersight-copilot-server

# Verify
curl -s http://127.0.0.1:8006/health

Troubleshooting

Symptom	Check
`docker compose up` fails immediately	`sudo docker compose logs opersight-ai-fastapi --tail=30`
FastAPI health returns `agent_enabled: false`	`OPENAI_API_KEY` missing or blank in env file
`curl 8006` connection refused	Container not started; check `docker ps`
502 from Nginx	Service not listening on 8006/8007; run `ss -tlnp \\| grep 800`
CORS error in browser	Check `CORS_EXTRA` in copilot-server env; check OPTIONS response headers
`Root_PATH` wrong in Swagger	Confirm `ROOT_PATH=/opersight/api/fastapi` in FastAPI env file
`AGUI_AGENT_URL` not resolving	Must use Docker service name (`opersight-ai-fastapi`), not `localhost`; both containers must be on same network

Extending to swmm and flooding

Repeat this entire document, replacing:

Placeholder	swmm	flooding
`opersight`	`swmm`	`flooding`
Port 8006	8004	8008
Port 8007	8005	8009
`opersight_agent`	`swmm_agent`	`flooding_agent`
`/opersight/api/fastapi`	`/swmm/api/fastapi`	`/flooding/api/fastapi`

The Docker images (hydrotrek-ai-fastapi:latest, hydrotrek-copilot-server:latest) are already built after Opersight is deployed. For additional suites, skip the repo clone and image build steps unless the shared repos changed. Start with the directory/env/compose steps for the new app.

README.md update (workstation)

Add two rows to the Deployed APIs table in ../README.md:

Service	Repo	Container port	Host (`127.0.0.1`)	Nginx (HTTPS)	Status
`opersight-ai-fastapi`	`repos/hydrotrek-ai-fastapi`	8080	8006	`/opersight/api/fastapi/`	this doc
`opersight-copilot-server`	`repos/hydrotrek-copilot-server`	8082	8007	`/opersight/api/copilotkit`	this doc

Deploy opersight suite

Architecture​

Port allocation​

Prerequisites​

Step 1 — Add Dockerfile to hydrotrek-ai-fastapi​

Step 2 — Create Opersight template files​

templates/docker/hydrotrek-opersight-suite/docker-compose.yml​

templates/docker/hydrotrek-opersight-suite/docker-compose.prod.yml​

templates/docker/hydrotrek-opersight-suite/env/opersight-ai-fastapi.env.example​

templates/docker/hydrotrek-opersight-suite/env/opersight-copilot-server.env.example​

templates/nginx/opersight-gqc-addition.conf​

Step 3 — Server: Create directories​

Step 4 — Server: Set up deploy keys​

hydrotrek-ai-fastapi deploy key​

hydrotrek-copilot-server deploy key​

Step 5 — Server: Clone repos​

Step 6 — Server: Build Docker images​

Step 7 — Server: Create env files​

Step 8 — Server: Place compose file​

Step 9 — Server: Bring up containers​

Step 10 — Server: Add Nginx routes​

Step 11 — Server: Add systemd boot unit​

Step 12 — Verify the deployment​

Routine update procedure​

Troubleshooting​

Extending to swmm and flooding​

README.md update (workstation)​